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Effective analyses of performance for detection schemes that optimally 
decode digital data in the presence of intersymbol interference have been 
slow in coming. Recently, however, Forney has given an upper bound on 
the bit error probability for maximum-likelihood sequence estimation. 
Starting from a standard geometrical framework, we give a much simplified 
derivation of this upper bound. Our derivation places the validity of this 
important bound more in evidence in that the concepts of whitened matched 
filter and error event are not introduced. 

Let oy, j = 1, 2, • • ■, N, be independent, equilikely binary random 
variables taking values ±1. Data transmission usually involves 
estimating the a, from a pulse sequence of the form 

ZaMt-jT), -co <t < co, (1) 

3—1 

which is observed in white gaussian noise of (two-sided) spectral 
density No/2. In (1), the minimum assumption put on the pulse wave- 
form h(t) is that it be L 2 . One possible detection procedure is to decide, 
on the basis of the received noisy signal, which one of the 2 N equilikely 
signals given in (1) was "most likely" (maximum-likelihood sequence 
detection) and use the sequence \a„) which is associated with that 
sequence as the detected symbols. As N grows large, the probability of 
deciding incorrectly on the sequence approaches unity; however, the 
real question revolves about the bit-error probability for maximum- 
likelihood sequence detection. Important work on this problem was 
done recently by Forney, 1,2 who showed that, under certain conditions 
on hit), the bit-error probability for large signal-to-noise ratios goes 
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exponentially to zero as 

P.«(coefif.)exp(-^). N ->0, (2) 

where d 2 min is the minimum distance between the signal sequences in 
(1); i.e., if we add a superscript to distinguish among sequences thus 



«(0 - L afhit - jT), - oo < t < co » - 1, 2, • • -, 2* (3) 



then 



di 



= lim min [" | «<«>(«) -««(*)|Vt. (4) 

JV-+00 »•, A J -oo 



t, 



Forney's demonstration consists of two steps. First, a lower bound on 
P e of the form (2) is established, valid for any h(t). Second, if H{w) 
denotes the Fourier transform of h(t) and 

^- f \H(u))\ 2 e iku dw = for integer k, \k\ > v, v integer, (5) 

then an upper bound for P e can be given which is convergent for large 
signal-to-noise ratios and which, furthermore, also has an asymptotic 
form given by (2) . 

In our opinion, Forney's discussion of the upper bound is sufficiently 
complicated that some question remains as to how firmly the result is 
established. We shall give a much simpler derivation, but first let us 
review the situation when v = 0, i.e., when there is no intersymbol 
interference. Using the reduction to the standard geometrical picture, 3 
the signal points (sequences) received in the absence of noise are as 
shown in Fig. 1. These signal points are to be regarded as being per- 
turbed by spherically symmetric, iV-dimensional, zero mean gaussian 
noise ; the variance of each component of the noise is No/2. Thus any 
point in the iV-dimensional space may be received and the maximum- 
likelihood decoder chooses the unperturbed sequence nearest to the 
received point as the transmitted one. The decoding regions are shown 
in Fig. 2 by dashed lines, and are labeled Rij in an obvious way. 

Now assume we transmit (1,1) and ask for the probability that the 
first bit is in error. This is the same as the probability that the received 
signal point is in B_ U U R-u-u or equivalently that the received 
signal point is to the left of the line labeled S in Fig. 1. In iV dimensions 
it would be the probability that the received signal point is on the 
opposite side of an (N — 1) dimensional hyperplane. This is clearly a 
simple one-dimensional gaussian problem having the well-known Q- 
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Fig. 1 — Four signal points corresponding to sequences for N = 2 in the absence of 
intersymbol interference. 
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Fig. 2 — Four signal points with intersymbol interference. 
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function for an answer, independent (because of the simple geometry) 
of the dimension or of which bit in the sequence is transmitted. 

When intersymbol interference is present, the error probability for 
the fcth bit may well depend on A; and N. In addition, the surface S 
which separates the sequences that have a k = +1 from those which 
have a k = —1 is no longer a hyperplane, although it is made up of 
segments which are hyperplanes. Finally, and perhaps a bit vaguely, 
the "shape" of the surface may depend on k. An example f or N = 2 is 
given in Fig. 2 showing the separating surface for the first bit. 

Our goal is to derive Forney's upper bound by geometrical arguments 
about as simple as those used in the discussion of Fig. 1. 

As in (3), we consider signal points identified by their respective 
data sequences {di\ N and label them with a superscript. We focus 
on the /cth bit being in error, and define sets A and B : 

A = {a«>|a£° = +1}> B = {a (i) |«*i° = ~M 

^ {b<«}. (6) 

We are mainly interested in the chance that the maximum-likelihood 
decoder selects a point b£fi, given that a particular sequence a (1) 
(say), a (1) E A, was transmitted. One upper bound on this is the union 
bound 

bes \ V2iVo / 

where d(a (1) , b) is the euclidean distance between a (1) and b; i.e., if I 
labels the particular b sequence, 



<P(a (1) ,b) = r |s (,) (0 -i»(0 

J — go 



2 dt 



2tt ;_ 



\H(u) 






dco. (8) 



On writing (8), and henceforth, we set T = 1. Equation (7) is a bad 
bound because it includes too many terms on the right-hand side. This 
is easily seen by applying it to the N-dimensional hypercube (no inter- 
symbol interference), for which we obtain (ignoring unessential 
coefficients) 

P,(k) ^ 2"e-< d, ' 4 "° ) , (9) 

where d 2 is the length of a hypercube edge. Thus, the bound, for any 
fixed No, approaches » as the length of the sequence N increases. 

Our next step will be to make some simple observations about the 
geometry of the received signal points [when (5) is true] that will allow 
us to delete most of the terms on the right-hand side of (7). Following 
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Forney, we are motivated to define another set Bi of signal points 
which is a subset of B. A vector b E B\ if {b, — a^) jLi (after deleting 
all zeros which begin the sequence and end the sequence of coefficients 
|bj — ay }) does not contain v or more consecutive zeros either to the 
right of the kth position or to the left of the fcth position. Forney's 
upper bound then reads 

P e (k) ^ £ o( d(a ^L_ b) V (10) 

To see why no further terms need be included in (10), select an arbi- 
trary signal point b* not included in the sum in (10) ; i.e., b*E£ 
— Bi = B*. From the way things have been defined, we may write 

a^ = («"■, 1, «x, « 2 , a 3 ) (11) 

b* = (? L , -1, 0!,« 2 , ff,), (12) 

where ai — (3i does not contain v consecutive zeros in its coefficient 
sequence and a 3 — (3 3 9* 0. The 1 in (11) and the ( — 1) in (12) occur 
in the fcth position, and a 2 is at least of dimension v, corresponding to 
the v (or more) positions where a 1 and b* are to agree. Now ai, — @l 
may or may not contain v consecutive zeros,* and we distinguish these 
two cases in our discussion. First assume that ol — ?l does not contain 
v consecutive zeros. Then note that 

b<" = (&l, -1, hi «*> «a) EBi (13) 

a«> a ( ttL , 1, ttl , a 2 , g,) G A. (14) 

Statement (13) is true by the absence of v consecutive zeros in ($ L — aL 
and also in £$i — tt\\ (14) is true because a{- 2) — +1. We may at this 
point imagine the four signal points a (1) , b*, b (1) , a (2) in general position 
as in Fig. 3. Now focus attention on the triangle (a (1) , a (2) , 6*).* We 
have (letting 0, denote a string of at least i zeros) 

d 2 (a (1) , 6*) = l|a (1) - &*|| 2 

= ||a L - L , 2, ai - ft, 0„ a 3 - ^ 3 || 2 

= ||a L - |8 L , 2, a, - j8i||» + ||a 3 - 3 \\ 2 . (15) 

The last step in (15) follows from the Fourier integral expression (8) 
for the distance and from (5), the requirement that intersymbol inter- 
ference not extend beyond v. The right member of (15) can readily 
be seen from (11, 12) and (13, 14) to be d 2 (« (2) , &*) + ^(a^, a (2) ). 



f Recall that a string of zeros in the beginning does not count. 
1 We drop bold-face notation for vectors here, and also allow ourselves the freedom 
of writing the subscript L as a superscript. 
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Fig. 3 — The four signal points defined in the test illustrated in general position 
(not necessarily planar). 



Hence, 

d 2 (<z (1) , b*) = <P(a«\ b*) + d 2 (a (1) , a™) (16) 

and (a (1) , a (2) , b*) forms a right triangle. In an entirely similar manner, 
one verifies 

d 2( a (8) j b (D) = d 2( a (l) ; &<!>) + d 2( aa)j a (2)) j (17) 

implying that (a (1) , a (2) , b (1) ) is a right triangle. Since 6 (1) - b* = a a) 
— a (2) , we have 

cP(a™, a<") = <P(6 (1 >, &*) (18) 

and the additional fact that the four points lie in the same plane. 
Equations (16), (17), and (18), and planarity, imply that (a< 1} , a (2) , 
6 (1) , 6*) form a rectangle as shown in Fig. 4. This demonstration assumes 
that there are v consecutive zeros to the right of the fcth position and 
not to the left. If we interchange the words "right" and "left", the same 
type of demonstration will apply. There remains the case when there 
are v consecutive zeros both to the right and to the left of the fcth 
position. In this case we write 

a™ = {a\, a \, a \,\,a h « 2 ,ocz) (19) 

b* = (0if,<ri0i -l,0i, a., 0,), (20) 

where a 3 - j8 3 9* ^ a\ - &\ and neither an - 0i nor a\ - ft con- 
tain v consecutive zeros in their coefficients. Further, we are to assume 
a 2 and a\ each have dimension at least v. If we define 

6< l > = (a\ a\ ft, -1, /Si, « a , a,) G #i (21) 

a (2) = (ft, a\, a\, 1, ai, at, 0i) 6 A, (22) 
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the proof that (a (1) , a <2) , 6 (1) , b*) is a rectangle can be carried out using 
the same techniques as earlier. 

Figure 4 makes it clear why, if a (1) is transmitted, terms like b* do 
not have to be included in the right-hand side of (10). The term 



Q (dJa±V)\ 



in (10) is the probability that, if a (1) is transmitted, the received signal 
will be on the "wrong side" of the hyperplane H which perpendicularly 
bisects the line (b x a l ). Now b* only needs to be included in (10) if its 
associated decoding region contributes some set of points of positive 
measure not accounted for in some other way. Thus, from Fig. 4, this 
is only the case if these new points were on the same side of H as 
a (1) . But any point on that side is closer to a (2) than b* and, hence, 
would never be decoded into b*. Hence the gaussian measure of the 
decoding region for 6* is already included in the term 



If we calculate further upper bounds for (10) by letting N = °° , 
we obtain a bound independent of k. Averaging this over the possible 
transmitted symbols gives precisely Forney's upper bound. The fact 
that the resulting upper bound converges for N small enough (for 




Fig. 4 — The actual relationship of the four signal points defined in the text. 
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N = oo ) has been recently discussed by Foschini. 4 This last step is an 
important one in a full proof, and was overlooked in the initial work. 2 
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